EMPWC: Expectation Maximization with Particle Swarm Optimization based Weig- hted Clustering for Outlier Detection in Large Scale Data

Outlier detection is usually considered as a pre-processing step for locating in a data set, those objects that do not conform to well-defi ned notions of expected behaviour. It is very important in data mining for discovering novel or rare events, anomalies, vicious actions, exceptional phenomena etc. However, investigation of outlier detection for categorical data sets is especially a challenging task because of the diffi culty of defi ning a meaningful similarity measure. In addition, one might have to determine the optimal value of outliers, that is, how many outliers a data set really has. A possible theoretical approach to this problem is to search for a range of values of outliers (o) and decide on an optimal value of outlier (o) by optimizing certain variational property. Because of this reason, Particle Swarm Optimization (PSO) method is introduced in this paper to search for a range of values of outliers (o). The proposed work consists of three major steps: (i) defi ne a function for outlier factor (ii) optimization value of outlier and (iii) clustering methods for outlier detection. In the fi rst step of the work, defi ne a new concept of entropy that takes both Shannon and Jensen-Shannon Divergence (JSD) into consideration. The second step, PSO is introduced to search outliers. Here, the PSO includes n number of data samples N which are moving around a D-dimensional search space for optimizing a certain variational property. Based on this PSO, defi ne a function for the outlier factor of an object which is solely determined by the object itself and can be updated effi ciently. Finally, propose EMPWC outlier detection method which requires no user-defi ned parameters for deciding whether an object is an outlier. In addition to this EMPWC based outlier detection methods associate a weight from entropy function with each observed dataset samples. Here, introduce the weighted-data Gaussian mixture and EM algorithms. The fi rst one considers a weight for each categorical data attributes. The second one treats each weight and detects outliers. The experiment results on large scale categorical datasets demonstrate that the proposed EMPWC based outlier detection methods can achieve a better tradeoff between Detection Rate (DR), False Alarm Rate (FAR) when compareing to state-of-the-art outlier detection approaches.